Physical haptic feedback system with spatial warping

ABSTRACT

A computing system including a head mounted display device with a processor and an associated display is provided. A sensor in communication with the processor is configured to detect a movable body part of a user. A plurality of physical haptic feedback structures are configured to be contacted by the movable body part. The processor is configured to operate the display device, receive data from the sensor, and determine an intended virtual target of the movable body part and a target physical structure having haptic characteristics corresponding to the intended virtual target. Also, the processor is configured to compute a path in real three-dimensional space from the movable body part to the target physical structure, compute a spatial warping pattern, and display via the display the virtual space and the virtual reality representation according to the spatial warping pattern.

BACKGROUND

The evolution of virtual reality systems has primarily emphasized visualand auditory simulation, while the incorporation of haptics, or tactilesimulation, has lagged. Tactile simulation can provide the user withtangible feedback that augments a virtual image being presented to theuser. Without such tactile simulation, virtual reality technologies failto provide the user with as authentic an immersive experience as couldbe provided.

SUMMARY

A computing system for physical haptic feedback with spatial warping isprovided. The system may include a head mounted display device includinga processor and an associated display and a sensor in communication withthe processor, the sensor being configured to detect a movable body partof a user. The system may include a plurality of physical hapticfeedback structures configured to be contacted by the movable body part,the structures positioned at different respective positions in realthree dimensional (3D) space. The plurality of physical haptic feedbackstructures may include a first structure and a second structure, thefirst structure having haptic characteristics differentiable from thesecond structure.

The processor may be configured to operate the display device to displaya virtual 3D space corresponding to real 3D space, and receive from thesensor data indicating a detected location of the movable body partwithin real 3D space. The processor may be configured to operate thedisplay device to display a virtual reality representation of themovable body part, a position of the virtual representation of themovable body part being displayed so as to appear to be positioned in avirtual location within the virtual space corresponding to the detectedlocation in real 3D space.

The processor may be further configured to determine, from among aplurality of virtual targets in the virtual space and a detected motionof the movable body part, an estimated intended virtual target of themovable body part, and determine a target physical structure havinghaptic characteristics corresponding to the intended virtual target. Theprocessor may be further configured to compute a path in the real 3Dspace from the movable body part to the target physical structure andcompute a spatial warping pattern to warp an image displayed on thedisplay. Further, the processor may be configured to display via thedisplay the virtual space and the virtual reality representationaccording to the spatial warping pattern.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of a first implementation of physical hapticfeedback system with virtual warping, including a head mounted displaydevice configured to display a virtual 3D space to a user, and physicalhaptic feedback structures positioned in locations in real 3D spacecorresponding to the virtual 3D space.

FIG. 2 illustrates a second implementation of a physical haptic feedbacksystem with virtual warping, configured with a server in communicationwith the head mounted display device and other components of the system.

FIGS. 3A-D are schematic drawings illustrating the manner in which aspatial warping pattern is applied by the systems of FIGS. 1 and 2.

FIGS. 4A-4C illustrate thresholds applied to determine whether a spatialwarping pattern is to be applied, as well as retargeting of spatialwarping toward new physical target structures and virtual targets.

FIG. 5 is a graphical depiction of a spatial warping pattern applied bythe systems of FIGS. 1 and 2.

FIG. 6 is a schematic view of a haptic feedback system of the system ofFIGS. 1 and 2, formed with a convex base element.

FIGS. 7A-7C are schematic views of the haptic feedback system of FIG. 6,with determination of a target physical structure from among the hapticfeedback structures based upon proximity and similarity between thevirtual target and haptic feedback structures.

FIG. 8 is a flowchart of a method for use with a computing device for aphysical haptic feedback system with spatial warping according to oneimplementation of the present disclosure.

FIG. 9 is a schematic depiction of an example computer device that maybe used as the computing devices in the systems of FIGS. 1 and 2.

DETAILED DESCRIPTION

The inventors have recognized that a challenge associated withsimulating tactile experiences for users of virtual reality systems isthat it can be difficult to provide multiple types of haptic stimuli tothe user, particularly when attempting to provide these stimuli atlocations that correspond to locations of virtual objects in a virtualimage shown to the user. The inventors have conceived of a way toovercome this challenge by using a plurality of real-world hapticobjects that correspond to objects in the virtual environment, andthrough spatial warping of the image displayed to the user, redirectinga user's hand or other body part to contact an appropriate one of thephysical objects for haptic stimulation, while viewing a virtualrepresentation of their hand or other body part approaching andcontacting the corresponding virtual object. A user's hand reaching foran object in the virtual environment can therefore be directed to aspecific location in the real world, the display being warped to guide auser to a physical structure matching the intended haptic target of theuser in the virtual space, as explained in detail below.

FIG. 1 illustrates a computing system 10 for physical haptic feedbackwith spatial warping, according to one implementation of the presentdisclosure. The computing system 10 includes a haptic feedback system12, and a head mounted display device 14 including a processor 16 and anassociated display 18. The processor is configured to execute softwareprograms stored in memory 82, as described below, to display via thedisplay 18 a virtual 3D space 42 populated with virtual objects. Thehaptic feedback system 12 includes a plurality of physical hapticfeedback structures 26 configured to be contacted by a movable body part22 of a user 24. Under command of the programs executed by theprocessor, the image of the virtual 3D space shown to the user via thedisplay 18 of the head mounted display device 12 can be spatially warpedso that the user 24 is perceptually influenced to move the movable bodypart 22 to an appropriate haptic feedback structure 26, as described indetail below. Despite the spatial warping, the user 24 wearing the headmounted display device 14 in real 3D space 40 is able to interact withreal objects 41 and receive haptic feedback that the user perceives iscoming from the same location as the virtual objects in the virtual 3Dspace. By selectively applying spatial warping in this way, the programsin the HMD device 14 can guide the movable body part to an appropriateone of the plurality of haptic feedback structures 26 that provideshaptic feedback that is most tailored for a particular virtual objectwith which the user interacts.

In order to track the user's body orientation and anticipate userinteraction with the virtual objects, one or more sensors may be incommunication with the processor 16 via the input/output interface 80,the sensors being configured to detect location and motion of at leastone user 24 including at least one movable body part 22. Thus, the user24 is monitored by various sensors associated with the HMD device 14,including onboard sensors 20 within the HMD device 14 and externalsensors such as motion tracking device 70 and camera 72. Onboard sensors20 may include eye tracking sensors 23 to track the user's gazedirection and an inertial motion (IMU) unit 21 that functions as a headtracking sensor to track the user's head pose orientation. If eyetracking sensors 23 are absent, the general orientation of the HMD 14may be used. The IMU 21 may include inclinometers to monitor pitch, yawand roll; accelerometers to measure g-forces; and gyrometers todetermine angular velocity. The user's head pose orientation istypically represented as X, Y, Z position, and pitch, roll, and yawrotation within the real 3D space. A front facing camera 25 is providedthat may function as an onboard motion tracking sensor that tracks theuser's movable body part, such as a hand, arm, digit, leg, foot, etc.External motion tracking sensors may also be used to track the motion ofthe user's movable body part 22. A virtual target 46 may be inferredfrom initial ballistic motion and velocity of, for example, hand motion.A motion tracking device 70 that is worn on the wrist of the user 24 mayprovide hand tracking data. One or more external cameras 72 may beimplemented, as described below.

Camera 72 may be configured to sense visible and/or non-visible light,and may be configured as an RGB camera, depth camera, or stereo/multiplecameras, for example. When configured as a visible light camera, visiblelight may be sensed by a CMOS or other sensor and may be passed as imagedata to processor 16, which may apply image processing techniques on theimage data to recognize movement of the movable part of the body 22within the field of view of the camera 72. When configured as a depthcamera, depth data may be calculated from passive stereo depthestimation techniques or phase or gated time of flight techniques. Astructured light depth imaging approach may be employed, where, in oneimplementation, an illumination pattern of infrared light rays isprojected onto surfaces of the real 3D space 40. In this implementation,the depth camera is configured to receive the reflected structured lightand the processor 16 calculates depth data for the surfaces. The depthcamera may output a depth image, which for each pixel captured in theimage contains a depth value sensed by the depth camera. By capturingvisible light image data and depth image data in this manner, skeletaltracking techniques may be applied to identify the movable body part 22within the images, and track the position, motion, and gesturesperformed by the movable body part 22 within the real 3D space 40.

It should be understood that this list of sensors that can be used asonboard sensors 20 and external sensors is meant to be illustrative, notexhaustive. For example, optical tags may be affixed to the user'smovable body part, and an array of visible light sensors may beprovided, with processor 16 configured to recognize the position andmotion of the tags based on the position of the tag in the images outputby each sensor in the array. Additionally, the user 24 and the real 3Dspace 40 may be monitored by radar, lidar, acoustic sensors, RF beacons,or magnetic sensors, where the sensors may be wearable as well.

The input/output interface 80 of HMD display device 14 is configured toenable the HMD device 14 to communicate with remote devices such asremote computers, cameras, and accessory devices, typically via a wired(e.g., Ethernet or USB) or wireless (e.g. WIFI or BLUETOOTH) connection.A virtual reality program 84 and retargeting program 86 are stored innon-volatile memory (E.g., FLASH RAM) 82 of the HMD device 14 andexecuted using portions of volatile memory (E.g., RAM) of memory 82 byprocessor 16. The virtual reality program 84 and a retargeting program86 are executed by the processor 16, using portions of memory 82. Eachof the components of display device 14 are configured to exchange dataand communicate as described below.

The haptic feedback system 14 includes a plurality of physical hapticfeedback structures 26, which are configured to be contacted by themovable body part 22. The haptic feedback structures 26 are positionedat different respective positions in real 3D space 40. It will beappreciated that the physical haptic feedback structures 26 may bestationary objects, movable objects, or objects that include movableparts, and may be mounted to a continuous body such as such a panel 43.It will be appreciated that first, second and third types of structures28, 29, and 30 are depicted in FIG. 1, each having hapticcharacteristics that are differentiable from each other. One potentialadvantage of having a plurality of haptic feedback structures 26 withdifferentiable haptic characteristics is that a plurality of virtualobjects may be mapped to a first haptic feedback structure 28 whileconcurrently a plurality of virtual objects with different hapticcharacteristics may be mapped to a second haptic feedback structure 29,and so on. In the illustrated implementation, the first structure 28 isa button while the second structure 29 is a dial and the third structure30 is a switch. The illustrated types of haptic feedback structures arenot intended to be limiting, as a variety of types of structures may beutilized. For example, haptic feedback structures may be used thatfeature soft or hard surfaces, hot or cold surfaces, tacky or smoothsurfaces, etc.

As discussed above, the processor 16 is configured to operate the HMDdisplay device 14 to display a virtual 3D space 42 corresponding to real3D space 40. Furthermore, the processor 16 is configured so that it mayreceive data from the various sensors, including data that indicates adetected location of the movable body part 22 within the real 3D space40. By supplying data from onboard sensors 20 and external sensors viathe input/output interface 80, the accuracy of the system'sdetermination of the position of the movable body part may be enhanced.

The display device 14 is operated by the processor 16 to display avirtual reality representation 44 of the movable body part 22. Theposition of the virtual representation 44 of the movable body part 22 isdisplayed so as to appear to be positioned in a virtual location withinthe virtual 3D space 42 corresponding to the detected location in real3D space 40, at least prior to spatial warping. In one example, themovable body part may be object locked to the detected location of themovable body part 22, at least prior to the movable body part entering aregion to which a spatial warping pattern has been applied. When spatialwarping is applied, as discussed in detail below, the location of thevirtual representation of the movable body part in virtual 3D space andthe actual location of the movable body part in real 3D space may beseparated, and the paths of the movable body part 22 through the real 3Dspace 40 and the virtual 3D space 42 will diverge as shown in FIG. 1 bythe difference in positions at 22A, 44A. One potential advantage ofobject-locking the virtual representation 44 of the movable body part 22to the detected location of the movable body part 22 prior to applyingspatial warping is that the user's perception that the virtual movablebody part 22 is enhanced to feel more realistic by the correspondence inposition.

The processor 16 is further configured to determine, from among aplurality of virtual targets 46 in the virtual 3D space 42 and adetected motion of the movable body part 22, an estimated intendedvirtual target 48 of the movable body part 22. The intended virtualtarget 48 may be determined from user gaze direction, user gazeduration, movement of the movable body part 22 that may be projected ina vector direction, voice indication of the user 24, and/or theapplication context of the virtual reality program depicting the virtual3D space 42. Thus, for example, if the application context indicatesthat a virtual telephone is ringing in virtual 3D space 42, the user'sgaze is directed to the virtual telephone, the user is detected assaying “Let me answer,” and the user's movable body part 22 is detectedas beginning to move in a direction toward the virtual telephone, thenthe virtual reality program 84 is likely to determine that the user'sintended target is the virtual telephone, rather than a virtual coffeecup that is placed next to the virtual telephone.

It will be appreciated that after estimating an intended virtual target48, the haptic feedback system 12 via the processor 16 may determinefrom among the haptic feedback structures 26 a particular targetphysical structure 50 that has haptic characteristics corresponding tothe intended virtual target 48. The processor 16 may be configured todetermine to which of the plurality of physical haptic feedbackstructures 26 the movable body part 22 is to be directed based onvarious parameters including a distance between a current location ofthe movable body part 22 and the target physical structure 50, anorientation of the target physical structure 50 and the virtual targets46 in the virtual 3D space 42, or a haptic feedback mechanism in thetarget physical structure 50. Trajectory and velocity of the detectedmotion of the user 24 may influence this determination as well. Thus,for example, a target physical structure 50 may be determined from amongthe plurality of haptic feedback structures 26 based on which hapticfeedback structure 26 is closest to the movable body part 22. Further, aphysical haptic feedback structure 26 may be selected on the basis ofhaptic similarity to the intended virtual target 48, or selected basedon the extent to which a spatial warping pattern would have to beapplied. Another factor that may need consideration is the presence ofobstructions between the user 24 and the physical haptic feedbackstructures 26, which may be additional users. Since multiple factors areinvolved, these factors may be weighted and compared in making thedetermination.

Following this determination, the processor 16 is configured to computea path 52 in real 3D space 40 from the movable body part 22 to thetarget physical structure 50. The processor 16 is further configured tocompute a spatial warping pattern to warp an image displayed on thedisplay 18. Via the display 18, the processor 16 is configured todisplay the virtual 3D space 42 and the virtual reality representation44 according to the spatial warping pattern.

Once an estimated intended visual target 48 of the user 24 isdetermined, the spatial warping pattern is applied to the image of thevirtual 3D environment, warping a portion of it through which themovable body part 22 is being directed. As the user 24 perceives thevirtual representation 44 of the movable body part 22 moving through thewarped virtual 3D space 42, the haptic feedback system 12 guides theuser 24 to direct the movable body part 22 along path 52 to the targetphysical structure 50, while the virtual representation 44 of themovable body part 22 moves along a virtual path 53 to the intendedvirtual target 48. Accordingly, the virtual path 53 diverges from thepath 52. The user 24 may therefore interact with the virtual target 46while sensing haptic feedback from an appropriately selected targetphysical structure 50, without perceiving that the haptic feedbacksystem 12 has altered the path 52 of the movable body part 22 in orderto supply the appropriate haptic feedback.

In the illustrated implementation of FIG. 1, it was determined that theuser 24 intends to interact with an intended virtual target 48 that is aswitch that is set in an upward orientation; in response to thisanticipated interaction of user 24 with the virtual 3D space 42, thesystem 10 spatially warps a portion of the virtual 3D environment 42 toenable the user 24 to interact with a target physical structure 50 thatis a switch positioned in an upward position, similar to the estimatedintended virtual target 48.

As shown in FIG. 5, the spatial warping may be accomplished by applyinga spatial warping pattern 200. The spatial warping pattern may beconfigured as a vector field applied within a region of spatial warping.The vector field may operate to shift the position of the virtualrepresentation 44 of the movable body part 22 by the distance indicatedby a component vector in the field as the movable body part 22 passesthrough the region. The contour lines of such a vector field areillustrated in FIG. 5. In FIG. 5, the white dot represents the positionof a portion of the movable body part 22 in an original image that hasnot been warped. An object on the lower dot dashed line will be warpedbased on the size and direction of the vector underlying the lower dot.By applying spatial warping, the virtual representation 44 of themovable body part 22 may be warped to the location shown by the blackcircle. To avoid warping of the actual image of the virtualrepresentation 44 of the movable body part 22, only the central locationof the movable body part 22 is warped, and the virtual representation 44is placed with reference to the central location, as opposed to apixel-by-pixel warping which would distort the image. The spatialwarping pattern is applied to a region of real 3D space 40, and thus asthe user 24 disengages with the target physical structure 50, themovable body part 22 may pass through the spatial warping pattern again.The spatial warping pattern may be cancelled once the movable body part22 is removed from the region of spatial warping, and the hapticfeedback system 12 determines that the user 24 no longer intends tointeract with the intended virtual target 48.

FIG. 2 shows a second implementation of the computing system 10, inwhich the head mounted display device 14 may be communicativelyconnected to a server 74 via network 72. The network 72 may be a wiredor wireless, local or wide area network, for example. Server 74 includesa processor 78, memory, and an input/output interface 88. A virtualreality server program 84S and retargeting server program 86S are storedin non-volatile memory of memory 92 and executed by processor 78 usingportions of volatile memory of memory 92. In this implementation,external sensors such camera 72 and motion tracking device 70 areconfigured to communicate via the network 72 and the input/outputinterface 88 with the server 74. With this configuration, data from theexternal sensors is fed to the server 74, and the virtual reality serverprogram 84S is configured to compute the virtual 3D space in the mannerperformed by virtual reality program 84 in the first implementation. Theserver 74 may be configured to receive data from the HMD device 14indicating values detected by onboard sensors 20, may performcomputations for the virtual 3D space 42 based on this data and based onthe external sensor inputs it has received, and may then send thecomputed display data to the HMD device 14 via network 72 for display tothe user 24 via display 18. Since the HMD device 14 is portable and ispowered by one or more rechargeable batteries, it will be appreciatedthat by performing these computations on the server 74, battery life onthe HMD device 14 may be improved.

The spatial warping pattern is computed to redirect the movable bodypart 22 along the computed path 52 to the target physical structure 50.FIGS. 3A-3D depict the manner in which the spatial warping pattern maybe applied. The image warped by the spatial warping pattern may be arelevant portion of the image of the virtual 3D space 42, through whichthe movable body part 22 passes. FIG. 3A illustrates this application ofimage warping. In the virtual 3D space 42, the user 24 observes thevirtual reality representation 44 of a movable body part 22, which inthis instance is the user's hand. The user 24 directs her hand toward avirtual target 46, the sensors recording the movement and the processor16 determining the estimated intended virtual target 48 and a targetphysical structure 50. As shown in FIG. 3A, the target physicalstructure 50 in real 3D space 40 is not aligned with the estimatedintended virtual target 48 in virtual 3D space 42. If no spatial warpwere applied, the movable body part 22 and virtual realityrepresentation 44 would not be able to traverse the actual path 52 andvirtual path 53 concurrently. In FIG. 3B, a spatial warping pattern isapplied to the image of the virtual 3D space 42, as shown in FIG. 3B.Via the display 18, the processor 16 warps the image of the virtual 3Dspace 42 as shown to the user 24 such that the target physical structure50 and the estimated virtual target 48 as observed by the user 24 align.This alignment is based on the calculation of path 52 by the processorbetween the movable body part 22 and the target physical structure 50.The warp effect (i.e., distance of warp) due to the warping of thebackground image of the virtual 3D space 42 is shown graphically toincrease as the movable body part 22 approaches the target physicalstructure 50. It will be appreciated that since the background image iswarped, the virtual reality representation 44 of the intended virtualtarget 48 has shifted in position downward to the location of the targetphysical structure 50. Warping the background image may also be referredto as “world warping.”

Alternatively, as shown in FIG. 3C, the image warped by the spatialwarping pattern may be the virtual reality representation 44 of themovable body part 22. In this case, rather than warping the backgroundimage of the virtual 3D space 42, the foreground image of the movablebody part 22 is warped. It will be appreciated that since the foregroundimage (i.e., the virtual representation 44 of the movable body part 22)is shifted, the virtual reality representation 44 is shown travelingstraight toward the intended virtual target 48, rather than downwardtoward the target physical structure 50.

In this case, it is determined that the user 24 must direct her hand tothe target physical structure 50 that is currently aligned with adifferent virtual target 46 than the estimated intended virtual target48. Consequently, a spatial warping pattern is applied to the virtualreality representation 44 of the user's hand, as shown in FIG. 4B. Viathe display 18, the processor 16 warps the virtual realityrepresentation 44 as shown to the user 24 such that the user 24 observesher hand in the virtual 3D space 42 to be a distance from the estimatedintended virtual target 48 that coincides with a distance to the targetphysical structure 50 in real 3D space 40. This alignment is based onthe calculation of path 52 by the processor between the movable bodypart 22 and the target physical structure 50.

FIG. 3D illustrates that a combination of spatial warping of the virtualreality representation 44 of the movable body part 22 and of thebackground image of the virtual 3D space 42 may be applied. This may beuseful to minimize user perception that elements in either thebackground or foreground image have been spatially warped, for example.Warping of both the image of the virtual 3D space 42 and the virtualreality representation 44 of the movable body part 22 with a spatialwarping pattern can reduce the amount by which either image is warped,thereby avoiding excessive image distortions or perception by the user24 of misalignment between the virtual 3D space 42 and the real 3D space40.

As shown in FIG. 4B, it will be appreciated that the spatial warpingpattern may be dynamically recalculated by the processor 16 in a seriesof time steps, illustrated as T1-T4, based on dynamic determination ofthe intended target 48 of the movable body part 22. The retargetingprogram 86 executed by the processor 16 determines, as shown in FIG. 4B,that the user 24 intends to interact with an updated virtual target 48Arather than the original determined intended virtual target 48. In thiscase, the retargeting program 86 is configured to determine a virtualpath 53 to the updated virtual target 48A, and determine an updatedphysical path 52A to an updated target physical structure 50A.Therefore, the movable body part 22 is redirected dynamically to theupdated target physical structure 50A. As a result, after moving towardthe targets, the movable body part 22 contacts the target physicalstructure 50 concurrently with the virtual reality representation 44 ofthe movable body part 22 appearing to contact the intended virtualtarget 48. As shown in FIG. 4C, the dynamic retargeting implemented bythe retargeting program 86 may also select an updated target physicalstructure 50A even when the intended virtual target 48 remains the same.For example, the intended virtual target 48 may have changed state inthe game from a COLD button to a HOT button. In this case, theretargeting program 86 may re-compute an updated physical path 52A tothe nearest physical target structure 50 that features the HOT hapticcharacteristic associated with the updated virtual target state.

Turning now to FIG. 4A, prior to applying the spatial warping pattern,the processor 16 may be configured to determine whether or not a spatialwarping pattern is to be applied to the haptic feedback system 12, toavoid a situation in which the spatial warping is perceivable by theuser 24. This determination may be made based on a threshold distance Dbetween the intended virtual target 48 and the target physical structure50. For example, the distance D between the intended virtual target 48and the target physical structure 50 may be examined to determinewhether it exceeds a threshold T, which may be a linear function basedon the distance from the current position of the movable body part 22,as shown. If the threshold T is exceeded, then the haptic feedbacksystem 12 may determine not to apply the spatial warping pattern, sincewarping beyond the threshold runs the risk of being perceivable by theuser 24, and if the threshold is not exceeded, then the spatial warpingpattern may be applied. When computing spatial warping patterns forvirtual targets 46 of the user, there may be a plurality of possiblepaths 52 to the target physical structures 50 that are within thethreshold depicted in FIG. 4A. In such cases the spatial warping patternmay be computed to be minimized, thereby minimizing an amount by whichthe image displayed is warped.

As briefly discussed in relation to FIG. 4C, it will be appreciated thatone or more of a plurality of physical haptic feedback structures 26 maybe dynamically mapped to a plurality of virtual targets 46 in thevirtual 3D space 42. A user 24 may be directed to any one of a pluralityof target physical structures 50 based on various factors as describedabove, and a plurality of virtual targets 46 may be mapped to a singletarget physical structure 50. The mapping between virtual targets 46 andtarget physical structures 50 may, however, be altered by the computingsystem 10 based on the given virtual 3D space 42 and the factorspreviously described. Similar to the above discussion, the detectedmotion of the movable body part 22 by the onboard sensors 20 or externalsensor inputs is used by the processor 16 to determine, of the pluralityof virtual targets 46 in the virtual 3D space 42, the estimated intendedvirtual target 48. The movable body part 22 is then directed to one of aplurality of physical haptic feedback structure 26 that have a hapticcharacteristic corresponding to the intended virtual target 48, based onthis determination.

Retargeting may be implemented by calculation using several variables. Apoint P_(v) in the virtual 3D space 42 may be mapped to a physical proxypoint P_(p) so that for an offset T_(f)

T _(f) =P _(v) −P _(p).

If H_(p) is the physical position of the movable body part 22 and H₀ isa fixed point,

D _(s) =|H _(p) −H ₀ |, D _(p) =|H _(p) −P _(p)|.

A gradual offset W may be added to the position of the virtualrepresentation 44 using a shift ratio α:

${W = {\alpha \; T_{f}}},{\alpha = {\frac{D_{s}}{D_{s} + D_{p}}.}}$

At the beginning of the motion, the shift ratio has a value of 0 whileat full offset the shift ratio is 1, when the movable body part 22touches the target physical structure 50 in conjunction with the virtualrepresentation 44 appearing to touch the intended virtual target 48.Retargeting is accomplished by interpolation between the currentretargeting and the updated retargeting to the new target:

W=αT _(f)+(1−α)T ₀

where T₀ is the original offset. In a frame where a new touch target isdetermined, H₀=H_(p).

FIG. 6 is a schematic view of a haptic feedback system 12 formed by aconvex base element 201 having a continuous convex surface formed of aplurality of subplanes, with many of the subplanes including a hapticfeedback structure mounted thereto. As shown, a first type of hapticfeedback structure A (rotating knob) and a second type of hapticfeedback structure B (push button) are positioned throughout the convexsurface. A haptic feedback system 12 of this construction may be useful,for example, in cockpit simulations implemented by a flight simulatorvirtual reality program. By positioning the first and second types ofhaptic feedback structures throughout the convex surface, the feedbacksystem 12 can select from among multiple candidate structures whencomputing a path 52 to a particular target physical structure 50 for auser 24. As shown in FIGS. 7A-7C, to choose a target physical structure50, the feedback system 12 may first examine the proximity between eachcandidate haptic feedback structure 26, either by line of sight (FIG.7A) or closest distance (FIG. 7B), and also may look at the similaritybetween the virtual target 46 and the physical candidates by (1) type ofhaptic characteristic (e.g., rotating knob vs. push button) and (2)similarity of orientation (surface normal similarity between the surfacenormal of the virtual target 205 and the surface normal of the candidatehaptic feedback structure—see FIG. 7C). Presuming the virtual target 46is displaying an image of a push button A, the haptic feedback structure202 would be rejected for consideration by the feedback system 12, sinceit is the incorrect type of haptic characteristic, while haptic feedbackstructures 203 and 204 would remain under consideration for targetphysical structures 50. A comparison such as discussed above would beapplied that weighed the closer distance between the virtual target 205and haptic feedback structure 203 against the closer similarity insurface normal orientation between virtual target 205 and hapticfeedback structure 204. The comparison logic could be applicationspecific, such that a developer could adjust the weighting to suit theneeds of the specific application.

A score may be calculated to make the determination of a target physicalstructure 50 from among the haptic feedback structures 26. Distance tothe target physical structure 50, similarity between the intendedvirtual target 48 and the target physical structure 50, and orientationof a surface normal of the target physical structure 50 may, among otherfactors, be considered in the score. For example, within a group ofhaptic feedback structures 26 having a small shift ratio α there may bea particular feedback structure that is farther away but has a texturematching the intended virtual target 48. Also, in this example theparticular feedback structure may have a 35° surface normal over anotherone with a 45° surface normal, which more closely matches the surfacenormal of the intended virtual target 48. Consequently, the hapticfeedback structure 26 with these characteristics may be given a highscore as a potential target physical structure 50.

In addition, a dynamic haptic adjustment mechanism may be employed inthe haptic feedback system 12. Given that physical haptic feedbackstructures 26 may have a plurality of haptic characteristics, each ofwhich may be variable, a dynamic haptic adjustment mechanism may adjustthe haptic characteristics of the physical haptic feedback structures26. Possible haptic characteristics include, but are not limited to,applied force, pressure, rotation, rotatability, mechanical resistance,vibration, deformability (e.g., hardness or easy compressibility),elasticity, material texture (e.g., smoothness or roughness),temperature, electrical charge, electrical resistance, pressure fromvented air (non-contact), and emitted ultrasound (non-contact).

Alterations by the dynamic haptic adjustment mechanism may includealtering the haptic characteristic itself, for example controlling ahaptic feedback structure 26 to rotate instead of vibrate.Alternatively, the dynamic haptic adjustment mechanism may adjust theintensity of the haptic characteristic. A physical haptic feedbackstructure 26 emitting heat may be controlled to decrease or increase theamount of heat emitted by the dynamic haptic adjustment mechanism. Inthe example of FIG. 4C, it will be appreciated that two physical buttonsare provided, one hot and one cold; however, alternatively a dynamichaptic adjustment mechanism may be provided in the form of a singlebutton that may switch from a hot state to a cold state. The adjustmentitself may be determined based on a specific virtual realityimplementation in the virtual reality system. Alternatively, theadjustment may depend on user interaction with features of the virtual3D space 42. In one implementation, a user 24 depressing a button thatappears as a virtual target 48 in a vehicle in the virtual 3D space 42to maneuver towards a heat source may subsequently feel heat fromanother haptic feedback structure 26. In this way, dynamic adjustment ofthe haptic characteristics may be used to provide dynamic hapticexperiences that match the content or state of the virtual realityprogram 84.

Furthermore, a haptic characteristic may be altered by the dynamichaptic adjustment mechanism according to determination of the intendedvirtual target 48 of the movable body part 22. In one implementation,should a user 24 reach for an estimated intended virtual target 48 thatrepresents a brake control in a virtual vehicle, she may be directed toa target physical structure 50 that may alter its mechanical resistancebased on whether it functions as a brake or as an accelerator. Inanother implementation, a user 24 that touches a haptic feedbackstructure 26 in one area may feel a vibration only from that location onthe haptic feedback structure 26. Also possible through theimplementation of redirection is guiding a user 24 with a portableobject. Based on the detected target destination of a portable objectheld or carried by the user 24, the processor 16 via the display 18 maydirect the user to place the object in a particular location using theapplication of a spatial warping pattern.

As some examples, the target physical structures 50 may include ahandle, dial, knob, button, switch, toggle, wheel, lever, pedal, pull,key, joystick, adjuster, or a touchpad. Alternatively, the targetphysical structures 50 may include tools, utensils, training equipment,or other objects appropriate to the specific implementation of thevirtual reality system. In one implementation, the physical hapticfeedback structures 26 may be surfaces formed as regions on a continuoussurface of a base material, such as a support element. In thisconfiguration, one continuous haptic surface has regions ofdifferentiable haptic characteristics. In one implementation, a firstsurface or region on the continuous surface may radiate heat while asecond surface or region on the continuous surface may vibrate. Inanother implementation, the continuous surface may be divided intosub-panels having different angular orientations relative to each other.For example, the continuous surface of the support element may be formedof a connected group of sub-planes generally approximating a curvearound a central zone within which a user stands when using the system.

FIG. 8 is a flow chart of a method for use with a computing device, fora physical haptic feedback system with spatial warping. Method 100 maybe executed using the systems described above, or utilizing othersuitable hardware and software elements.

At 102, the method includes operating a head mounted display device 14including a processor 16 and an associated display 18 to display avirtual 3D space 42 corresponding to real 3D space 40, the displaydevice 14 including onboard sensors 20 in communication with a processor16. At least one sensor is configured to detect a movable body part 22of a user 24.

At 104, the method further includes receiving from the sensor dataindicating a detected location of the movable body part 22 within real3D space 40. The method at 106 may further include operating the displaydevice 14 to display a virtual reality representation 44 of the movablebody part 22. The position of the virtual representation 44 of themovable body part 22 appears to be in a virtual location within thevirtual 3D space 42 corresponding to the detected location in real 3Dspace 40.

The method at 108 may further include determining, from among aplurality of virtual targets 46 in the virtual 3D space 42 and adetected motion of the movable body part 22, an estimated intendedvirtual target 48 of the movable body part 22. At 110, the method mayfurther include determining a target physical structure 50 having hapticcharacteristics corresponding to the intended virtual target 48, thetarget physical structure 50 being selected from among a plurality ofphysical haptic feedback structures 26 configured to be contacted by themovable body part 22, the structures 26 positioned at differentrespective positions in real 3D space 40, the plurality of physicalhaptic feedback structures 26 including a first structure and a secondstructure, the first structure having haptic characteristicsdifferentiable from the second structure.

At 112, the method may further include computing a path 52 in the real3D space 40 from the movable body part 22 to the target physicalstructure 50. The method at 114 may further include computing a spatialwarping pattern to warp an image displayed on the display 18. The methodat 116 may further include displaying via the display 18 the virtual 3Dspace 42 and the virtual reality representation 44 according to thespatial warping pattern.

As described above, the processor 16 may be configured to determine towhich of the plurality of physical haptic feedback structures 26 themovable body part 22 is to be directed. This determination may be basedupon a distance between a current location of the movable body part 22and the target physical structure 50. An orientation of the targetphysical structure 50 and the virtual target 48 in the virtual 3D space42 and a haptic feedback mechanism in the target physical structure 50may also be used in the determination. A clear path in the real 3D space40 to the target physical structure 50 is also a factor. The spatialwarping pattern may be computed to redirect the movable body part 22along the computed path 52 to the target physical structure 50. This maybe accomplished by warping an image of the virtual 3D space 42 accordingto the spatial warping pattern; alternatively, an image of the virtualreality representation 44 of the movable body part 22 may be warped byway of the spatial warping pattern. It will be appreciated that bothimages or a combination of images may be warped as well.

As further described above, the spatial warping pattern may be computedto redirect the movable body part 22 along the computed path 52 to thetarget physical structure 50, where the processor 16 is furtherconfigured to dynamically recalculate the spatial warping pattern in aseries of time steps based on dynamic determination of the intendedtarget 48 of the movable body part 22. Therefore, redirection of themovable body part 22 to the target physical structure 50 may be dynamic.Optimally the movable body part 22 will contact the target physicalstructure 50 concurrently with the virtual reality representation 44 ofthe movable body part 22 appearing to contact the intended virtualtarget 48. The path 52 may be one of a plurality of possible paths tothe target physical structure 50. Computation of the spatial warpingpattern may include computing a minimized spatial warping pattern thatminimizes an amount by which the image displayed is warped. Applicationof the spatial warping pattern, based upon a threshold distance betweenthe intended virtual target 48 and the target physical structure 50, mayalso be executed via the processor 16 as specified above.

As also described above, at least one of the plurality of physicalhaptic feedback structures 26 may be dynamically mapped to a pluralityof virtual targets 46 in the virtual 3D space 42. The processor 16 maydetermine the estimated intended virtual target 48 of the movable bodypart 22 from among the plurality of virtual targets 46 in the virtual 3Dspace 42. Based on the detected motion of the movable body part 22 andthe estimated intended virtual target 48, the movable body part 22 maybe directed to a physical haptic feedback structure 26.

As described above, via a dynamic haptic adjustment mechanism, at leasta first haptic characteristic of the physical haptic feedback structuresmay be adjusted. Haptic characteristics may include applied force,pressure, rotation, rotatability, mechanical resistance, vibration,deformability (e.g., hardness or easy compressibility), elasticity,texture (e.g., smoothness or roughness), temperature, electrical charge,electrical resistance, pressure from vented air (non-contact), andemitted ultrasound (non-contact). It will be appreciated thatalternatively the physical haptic feedback structures 26 may beimplemented to include a first surface and a second surface formed asregions on a continuous surface of a base material.

FIG. 9 schematically shows a non-limiting embodiment of an examplecomputing system 900 that can enact one or more of the methods andprocesses described above. Example computing system 900 is shown insimplified form. Example computing system 900 may embody the computingsystem 10. Example computing system 900 may take the form of one or morepersonal computers, server computers, tablet computers, networkcomputing devices, mobile computing devices, mobile communicationdevices (e.g., smart phone), and/or other computing devices.

Example computing system 900 includes a logic processor 902, volatilememory 903, and a non-volatile storage device 904. Example computingsystem 900 may optionally include a display subsystem 906, inputsubsystem 908, communication subsystem 1000, and/or other components notshown in FIG. 9.

Logic processor 902 includes one or more physical devices configured toexecute instructions. For example, the logic processor may be configuredto execute instructions that are part of one or more applications,programs, routines, libraries, objects, components, data structures, orother logical constructs. Such instructions may be implemented toperform a task, implement a data type, transform the state of one ormore components, achieve a technical effect, or otherwise arrive at adesired result.

The logic processor may include one or more physical processors(hardware) configured to execute software instructions. Additionally oralternatively, the logic processor may include one or more hardwarelogic circuits or firmware devices configured to executehardware-implemented logic or firmware instructions. Processors of thelogic processor 902 may be single-core or multi-core, and theinstructions executed thereon may be configured for sequential,parallel, and/or distributed processing. Individual components of thelogic processor optionally may be distributed among two or more separatedevices, which may be remotely located and/or configured for coordinatedprocessing. Aspects of the logic processor may be virtualized andexecuted by remotely accessible, networked computing devices configuredin a cloud-computing configuration. In such a case, it will beunderstood that these virtualized aspects are run on different physicallogic processors of various different machines.

Non-volatile storage device 904 includes one or more physical devicesconfigured to hold instructions executable by the logic processors toimplement the methods and processes described herein. When such methodsand processes are implemented, the state of non-volatile storage device94 may be transformed—e.g., to hold different data.

Non-volatile storage device 904 may include physical devices that areremovable and/or built-in. Non-volatile storage device 94 may includeoptical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.),semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH memory, etc.),and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tapedrive, MRAM, etc.), or other mass storage device technology.Non-volatile storage device 904 may include nonvolatile, dynamic,static, read/write, read-only, sequential-access, location-addressable,file-addressable, and/or content-addressable devices. It will beappreciated that non-volatile storage device 904 is configured to holdinstructions even when power is cut to the non-volatile storage device904.

Volatile memory 903 may include physical devices that include randomaccess memory. Volatile memory 903 is typically utilized by logicprocessor 902 to temporarily store information during processing ofsoftware instructions. It will be appreciated that volatile memory 903typically does not continue to store instructions when power is cut tothe volatile memory 903. One example of volatile memory 903 is randomaccess memory (RAM).

Aspects of logic processor 902, volatile memory 903, and non-volatilestorage device 904 may be integrated together into one or morehardware-logic components. Such hardware-logic components may includefield-programmable gate arrays (FPGAs), program- andapplication-specific integrated circuits (PASIC/ASICs), program- andapplication-specific standard products (PSSP/ASSPs), system-on-a-chip(SOC), and complex programmable logic devices (CPLDs), for example.

The terms “module,” “program,” and “engine” may be used to describe anaspect of example computing system 900 that is typically software storedin non-volatile memory and implemented by a processor to perform aparticular function using portions of volatile memory, which functioninvolves transformative processing that specially configures theprocessor to perform the function. Thus, a module, program, or enginemay be instantiated via logic processor 902 executing instructions heldby non-volatile storage device 904, using portions of volatile memory903. It will be understood that different modules, programs, and/orengines may be instantiated from the same application, service, codeblock, object, library, routine, API, function, etc. Likewise, the samemodule, program, and/or engine may be instantiated by differentapplications, services, code blocks, objects, routines, APIs, functions,etc. The terms “module,” “program,” and “engine” may encompassindividual or groups of executable files, data files, libraries,drivers, scripts, database records, etc.

When included, display subsystem 906 may be used to present a visualrepresentation of data held by non-volatile storage device 904. Thevisual representation may take the form of a graphical user interface(GUI). As the herein described methods and processes change the dataheld by the non-volatile storage device, and thus transform the state ofthe non-volatile storage device, the state of display subsystem 906 maylikewise be transformed to visually represent changes in the underlyingdata. Display subsystem 906 may include one or more display devicesutilizing virtually any type of technology. Such display devices may becombined with logic processor 902, volatile memory 903, and/ornon-volatile storage device 904 in a shared enclosure, or such displaydevices may be peripheral display devices.

When included, input subsystem 908 may comprise or interface with one ormore user-input devices such as a keyboard, mouse, touch screen,microphone, camera, or game controller. When included, communicationsubsystem 1000 may be configured to communicatively couple variouscomputing devices described herein with each other, and with otherdevices. Communication subsystem 1000 may include wired and/or wirelesscommunication devices compatible with one or more differentcommunication protocols. As non-limiting examples, the communicationsubsystem may be configured for communication via a wireless telephonenetwork, or a wired or wireless local- or wide-area network. In someembodiments, the communication subsystem may allow example computingsystem 900 to send and/or receive messages to and/or from other devicesvia a network such as the Internet.

According to the haptic feedback system 12 as described, hapticsimulation in a virtual 3D space 42 with a plurality of virtual targets46 may be provided by way of multiple physical haptic feedbackstructures 26 that are mapped to the virtual targets 46. By way of thishaptic feedback system 12, a user 24 can be directed to specificphysical objects corresponding to perceived virtual objects in thevirtual 3D space 42. Multiple physical objects can be used for thephysical haptic feedback structures 26, allowing for a broader range ofhaptic simulation. Additionally, each physical haptic feedback structure26 can be mapped to multiple virtual targets 46, making it possible tosimulate an even greater range of haptic experience. Redirection of theuser 24 is key to implementation of the haptic feedback system 12 withas great a variety of haptic simulation as possible.

The following paragraphs provide additional support for the claims ofthe subject application. One aspect provides a computing system,comprising a head mounted display device including a processor and anassociated display; a sensor in communication with the processor, thesensor being configured to detect a movable body part of a user; and aplurality of physical haptic feedback structures configured to becontacted by the movable body part, the structures positioned atdifferent respective positions in real three-dimensional space, theplurality of physical haptic feedback structures including a firststructure and a second structure, the first structure having hapticcharacteristics differentiable from the second structure. The processormay be configured to operate the display device to display a virtualthree-dimensional space corresponding to real three-dimensional space;receive from the sensor data indicating a detected location of themovable body part within real three-dimensional space; and operate thedisplay device to display a virtual reality representation of themovable body part, a position of the virtual representation of themovable body part being displayed so as to appear to be positioned in avirtual location within the virtual space corresponding to the detectedlocation in real three-dimensional space. The processor may beconfigured to determine, from among a plurality of virtual targets inthe virtual space and a detected motion of the movable body part, anestimated intended virtual target of the movable body part; determine atarget physical structure having haptic characteristics corresponding tothe intended virtual target; compute a path in the realthree-dimensional space from the movable body part to the targetphysical structure; compute a spatial warping pattern to warp an imagedisplayed on the display; and display via the display the virtual spaceand the virtual reality representation according to the spatial warpingpattern.

In this aspect, additionally or alternatively, the processor may beconfigured to determine to which of the plurality of physical hapticfeedback structures the movable body part is to be directed based uponat least one parameter selected from the group consisting of a distancebetween a current location of the movable body part and the targetphysical structure, an orientation of the target physical structure andthe virtual target in the virtual space, and a type of haptic feedbackmechanism in the target physical structure.

In this aspect, additionally or alternatively, the spatial warpingpattern may be computed to redirect the movable body part along thecomputed path to the target physical structure and the image warped bythe spatial warping pattern may be at least one of the group consistingof an image of the virtual space and an image of the virtual realityrepresentation of the movable body part.

In this aspect, additionally or alternatively, the spatial warpingpattern may be computed to redirect the movable body part along thecomputed path to the target physical structure, and the processor may befurther configured to dynamically recalculate the spatial warpingpattern in a series of time steps based on dynamic determination of theintended target of the movable body part, therefore causing redirectionof the movable body part to the target physical structure to be dynamicand the movable body part to contact the target physical structureconcurrently with the virtual reality representation of the movable bodypart appears to contact the intended virtual target.

In this aspect, additionally or alternatively, the path may be one of aplurality of possible paths to the target physical structure, andcomputation of the spatial warping pattern may include computing aminimized spatial warping pattern that minimizes an amount by which theimage displayed is warped. In this aspect, additionally oralternatively, the processor may be further configured to determineapplication of the spatial warping pattern based upon a thresholddistance between the intended virtual target and the target physicalstructure.

In this aspect, additionally or alternatively, at least one of theplurality of physical haptic feedback structures may be dynamicallymapped to a plurality of virtual targets in the virtual space and themovable body part may be directed to the physical haptic feedbackstructure based on the determination by the processor, from among theplurality of virtual targets in the virtual space and the detectedmotion of the movable body part, of the estimated intended virtualtarget of the movable body part. In this aspect, additionally oralternatively, a dynamic haptic adjustment mechanism may adjust at leasta first haptic characteristic of the physical haptic feedbackstructures, the first haptic characteristic being at least one of thegroup consisting of applied force, pressure, rotation, rotatability,mechanical resistance, vibration, deformability, elasticity, texture,temperature, electrical charge, electrical resistance, pressure fromvented air (non-contact), and emitted ultrasound (non-contact).

In this aspect, additionally or alternatively, the target physicalstructure may be from the group consisting of a handle, a dial, a knob,a button, a switch, a toggle, a wheel, a lever, a pedal, a pull, a key,and a joystick. In this aspect, additionally or alternatively, thephysical haptic feedback structures may include a first surface and asecond surface formed as regions on a continuous surface of a basematerial.

Another aspect provides a method for use with a computing device,comprising, at a processor, operating a head mounted display deviceincluding a processor and an associated display to display a virtualthree-dimensional space corresponding to real three-dimensional space,the display device including a sensor in communication with theprocessor, the sensor being configured to detect a movable body part ofa user; receiving from the sensor data indicating a detected location ofthe movable body part within real three-dimensional space; operating thedisplay device to display a virtual reality representation of themovable body part, a position of the virtual representation of themovable body part being displayed so as to appear to be positioned in avirtual location within the virtual space corresponding to the detectedlocation in real three-dimensional space; determining from among aplurality of virtual targets in the virtual space and a detected motionof the movable body part, an estimated intended virtual target of themovable body part; determining a target physical structure having hapticcharacteristics corresponding to the intended virtual target, the targetphysical structure being selected from among a plurality of physicalhaptic feedback structures configured to be contacted by the movablebody part, the structures positioned at different respective positionsin real three-dimensional space, the plurality of physical hapticfeedback structures including a first structure and a second structure,the first structure having haptic characteristics differentiable fromthe second structure; computing a path in the real three-dimensionalspace from the movable body part to the target physical structure;computing a spatial warping pattern to warp an image displayed on thedisplay; and displaying via the display the virtual space and thevirtual reality representation according to the spatial warping pattern.

In this aspect, additionally or alternatively, the processor may beconfigured to determine to which of the plurality of physical hapticfeedback structures the movable body part is to be directed based uponat least one parameter selected from the group consisting of a distancebetween a current location of the movable body part and the targetphysical structure, an orientation of the target physical structure andthe virtual target in the virtual space, and a type of haptic feedbackmechanism in the target physical structure.

In this aspect, additionally or alternatively, the spatial warpingpattern may be computed to redirect the movable body part along thecomputed path to the target physical structure and the image warped bythe spatial warping pattern may be at least one of the group consistingof an image of the virtual space and an image of the virtual realityrepresentation of the movable body part.

In this aspect, additionally or alternatively, the spatial warpingpattern may be computed to redirect the movable body part along thecomputed path to the target physical structure, and the processor may befurther configured to dynamically recalculate the spatial warpingpattern in a series of time steps based on dynamic determination of theintended target of the movable body part, therefore causing redirectionof the movable body part to the target physical structure to be dynamicand the movable body part to contact the target physical structureconcurrently with the virtual reality representation of the movable bodypart appears to contact the intended virtual target.

In this aspect, additionally or alternatively, the path may be one of aplurality of possible paths to the target physical structure, andcomputation of the spatial warping pattern may include computing aminimized spatial warping pattern that minimizes an amount by which theimage displayed is warped. In this aspect, additionally oralternatively, the processor may be further configured to determineapplication of the spatial warping pattern based upon a thresholddistance between the intended virtual target and the target physicalstructure.

In this aspect, additionally or alternatively, at least one of theplurality of physical haptic feedback structures may be dynamicallymapped to a plurality of virtual targets in the virtual space and themovable body part may be directed to the physical haptic feedbackstructure based on the determination by the processor, from among theplurality of virtual targets in the virtual space and the detectedmotion of the movable body part, of the estimated intended virtualtarget of the movable body part. In this aspect, additionally oralternatively, a dynamic haptic adjustment mechanism may adjust at leasta first haptic characteristic of the physical haptic feedbackstructures, the first haptic characteristic being at least one of thegroup consisting of applied force, pressure, rotation, rotatability,mechanical resistance, vibration, deformability, elasticity, texture,temperature, electrical charge, electrical resistance, pressure fromvented air (non-contact), and emitted ultrasound (non-contact). In thisaspect, additionally or alternatively, the physical haptic feedbackstructures may include a first surface and a second surface formed asregions on a continuous surface of a base material.

Another aspect provides a computing system, comprising a head mounteddisplay device including a processor and an associated display; a sensorin communication with the processor, the sensor being configured todetect a movable physical object under direct control of a user; and aplurality of physical haptic feedback structures configured to becontacted by the movable object, the structures positioned at differentrespective positions in a real three-dimensional space, the plurality ofphysical haptic feedback structures including a first structure and asecond structure, the first structure having haptic characteristicsdifferentiable from the second structure. The processor may beconfigured to operate the display device to display a virtualthree-dimensional space corresponding to real three-dimensional space;receive from the sensor data indicating a detected location of themovable object within real three-dimensional space; operate the displaydevice to display a virtual reality representation of the movableobject, a position of the virtual representation of the movable objectbeing displayed so as to appear to be positioned in a virtual locationwithin the virtual space corresponding to the detected location in realthree-dimensional space; determine, from among a plurality of virtualtargets in the virtual space and a detected motion of the movableobject, an estimated intended virtual target of the movable object;determine a target physical structure having haptic characteristicscorresponding to the intended virtual target; compute a path in the realthree-dimensional space from the movable object to the target physicalstructure; compute a spatial warping pattern to warp an image displayedon the display; and display via the display the virtual space and thevirtual reality representation according to the spatial warping pattern.

It will be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated and/ordescribed may be performed in the sequence illustrated and/or described,in other sequences, in parallel, or omitted. Likewise, the order of theabove-described processes may be changed.

The subject matter of the present disclosure includes all novel andnon-obvious combinations and sub-combinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

1. A computing system, comprising: a head mounted display deviceincluding a processor and an associated display; a sensor incommunication with the processor, the sensor being configured to detecta movable body part of a user; and a plurality of physical hapticfeedback structures configured to be contacted by the movable body part,the structures positioned at different respective positions in realthree-dimensional space, the plurality of physical haptic feedbackstructures including a first structure and a second structure, the firststructure having haptic characteristics differentiable from the secondstructure; the processor configured to: operate the display device todisplay a virtual three-dimensional space corresponding to realthree-dimensional space; receive from the sensor data indicating adetected location of the movable body part within real three-dimensionalspace; operate the display device to display a virtual realityrepresentation of the movable body part, a position of the virtualrepresentation of the movable body part being displayed so as to appearto be positioned in a virtual location within the virtual spacecorresponding to the detected location in real three-dimensional space;determine, from among a plurality of virtual targets in the virtualspace and a detected motion of the movable body part, an estimatedintended virtual target of the movable body part; determine a targetphysical structure having haptic characteristics corresponding to theintended virtual target; compute a path in the real three-dimensionalspace from the movable body part to the target physical structure;compute a spatial warping pattern to warp an image displayed on thedisplay; and display via the display the virtual space and the virtualreality representation according to the spatial warping pattern.
 2. Thesystem of claim 1, wherein the processor is configured to determine towhich of the plurality of physical haptic feedback structures themovable body part is to be directed based upon at least one parameterselected from the group consisting of a distance between a currentlocation of the movable body part and the target physical structure, anorientation of the target physical structure and the virtual target inthe virtual space, and a type of haptic feedback mechanism in the targetphysical structure.
 3. The system of claim 1, wherein the spatialwarping pattern is computed to redirect the movable body part along thecomputed path to the target physical structure and the image warped bythe spatial warping pattern is at least one of the group consisting ofan image of the virtual space and an image of the virtual realityrepresentation of the movable body part.
 4. The system of claim 1,wherein the spatial warping pattern is computed to redirect the movablebody part along the computed path to the target physical structure, andwherein the processor is further configured to dynamically recalculatethe spatial warping pattern in a series of time steps based on dynamicdetermination of the intended target of the movable body part, thereforecausing redirection of the movable body part to the target physicalstructure to be dynamic and the movable body part to contact the targetphysical structure concurrently with the virtual reality representationof the movable body part appears to contact the intended virtual target.5. The system of claim 4, wherein the path is one of a plurality ofpossible paths to the target physical structure, and wherein computationof the spatial warping pattern includes computing a minimized spatialwarping pattern that minimizes an amount by which the image displayed iswarped.
 6. The system of claim 1, wherein the processor is furtherconfigured to determine application of the spatial warping pattern basedupon a threshold distance between the intended virtual target and thetarget physical structure.
 7. The system of claim 1, wherein at leastone of the plurality of physical haptic feedback structures isdynamically mapped to a plurality of virtual targets in the virtualspace and the movable body part is directed to the physical hapticfeedback structure based on the determination by the processor, fromamong the plurality of virtual targets in the virtual space and thedetected motion of the movable body part, of the estimated intendedvirtual target of the movable body part.
 8. The system of claim 1,further comprising a dynamic haptic adjustment mechanism that adjusts atleast a first haptic characteristic of the physical haptic feedbackstructures, the first haptic characteristic being at least one of thegroup consisting of applied force, pressure, rotation, rotatability,mechanical resistance, vibration, deformability, elasticity, texture,temperature, electrical charge, electrical resistance, pressure fromvented air (non-contact), and emitted ultrasound (non-contact).
 9. Thesystem of claim 1, wherein the target physical structure is selectedfrom the group consisting of a handle, a dial, a knob, a button, aswitch, a toggle, a wheel, a lever, a pedal, a pull, a key, and ajoystick.
 10. The system of claim 1, wherein the physical hapticfeedback structures include a first surface and a second surface formedas regions on a continuous surface of a base material.
 11. A method foruse with a computing device, comprising: at a processor: operating ahead mounted display device including a processor and an associateddisplay to display a virtual three-dimensional space corresponding toreal three-dimensional space, the display device including a sensor incommunication with the processor, the sensor being configured to detecta movable body part of a user; receiving from the sensor data indicatinga detected location of the movable body part within realthree-dimensional space; operating the display device to display avirtual reality representation of the movable body part, a position ofthe virtual representation of the movable body part being displayed soas to appear to be positioned in a virtual location within the virtualspace corresponding to the detected location in real three-dimensionalspace; determining from among a plurality of virtual targets in thevirtual space and a detected motion of the movable body part, anestimated intended virtual target of the movable body part; determininga target physical structure having haptic characteristics correspondingto the intended virtual target, the target physical structure beingselected from among a plurality of physical haptic feedback structuresconfigured to be contacted by the movable body part, the structurespositioned at different respective positions in real three-dimensionalspace, the plurality of physical haptic feedback structures including afirst structure and a second structure, the first structure havinghaptic characteristics differentiable from the second structure;computing a path in the real three-dimensional space from the movablebody part to the target physical structure; computing a spatial warpingpattern to warp an image displayed on the display; and displaying viathe display the virtual space and the virtual reality representationaccording to the spatial warping pattern.
 12. The method of claim 11,wherein the processor is configured to determine to which of theplurality of physical haptic feedback structures the movable body partis to be directed based upon at least one parameter selected from thegroup consisting of a distance between a current location of the movablebody part and the target physical structure, an orientation of thetarget physical structure and the virtual target in the virtual space,and a type of haptic feedback mechanism in the target physicalstructure.
 13. The method of claim 11, wherein the spatial warpingpattern is computed to redirect the movable body part along the computedpath to the target physical structure and the image warped by thespatial warping pattern is at least one of the group consisting of animage of the virtual space and an image of the virtual realityrepresentation of the movable body part.
 14. The method of claim 11,wherein the spatial warping pattern is computed to redirect the movablebody part along the computed path to the target physical structure, andwherein the processor is further configured to dynamically recalculatethe spatial warping pattern in a series of time steps based on dynamicdetermination of the intended target of the movable body part, thereforecausing redirection of the movable body part to the target physicalstructure to be dynamic and the movable body part to contact the targetphysical structure concurrently with the virtual reality representationof the movable body part appears to contact the intended virtual target.15. The method of claim 14, wherein the path is one of a plurality ofpossible paths to the target physical structure, and wherein computationof the spatial warping pattern includes computing a minimized spatialwarping pattern that minimizes an amount by which the image displayed iswarped.
 16. The method of claim 11, further comprising, via theprocessor, determining application of the spatial warping pattern basedupon a threshold distance between the intended virtual target and thetarget physical structure.
 17. The method of claim 11, wherein at leastone of the plurality of physical haptic feedback structures isdynamically mapped to a plurality of virtual targets in the virtualspace and the movable body part is directed to the physical hapticfeedback structure based on the determination by the processor, fromamong the plurality of virtual targets in the virtual space and thedetected motion of the movable body part, of the estimated intendedvirtual target of the movable body part.
 18. The method of claim 11,further comprising adjusting, via a dynamic haptic adjustment mechanism,at least a first haptic characteristic of the physical haptic feedbackstructures, the first haptic characteristic being at least one of thegroup consisting of applied force, pressure, rotation, rotatability,mechanical resistance, vibration, deformability, elasticity, texture,temperature, electrical charge, electrical resistance, pressure fromvented air (non-contact), and emitted ultrasound (non-contact).
 19. Themethod of claim 11, wherein the physical haptic feedback structuresinclude a first surface and a second surface formed as regions on acontinuous surface of a base material.
 20. A computing system,comprising: a head mounted display device including a processor and anassociated display; a sensor in communication with the processor, thesensor being configured to detect a movable physical object under directcontrol of a user; and a plurality of physical haptic feedbackstructures configured to be contacted by the movable object, thestructures positioned at different respective positions in a realthree-dimensional space, the plurality of physical haptic feedbackstructures including a first structure and a second structure, the firststructure having haptic characteristics differentiable from the secondstructure; the processor configured to: operate the display device todisplay a virtual three-dimensional space corresponding to realthree-dimensional space; receive from the sensor data indicating adetected location of the movable object within real three-dimensionalspace; operate the display device to display a virtual realityrepresentation of the movable object, a position of the virtualrepresentation of the movable object being displayed so as to appear tobe positioned in a virtual location within the virtual spacecorresponding to the detected location in real three-dimensional space;determine, from among a plurality of virtual targets in the virtualspace and a detected motion of the movable object, an estimated intendedvirtual target of the movable object; determine a target physicalstructure having haptic characteristics corresponding to the intendedvirtual target; compute a path in the real three-dimensional space fromthe movable object to the target physical structure; compute a spatialwarping pattern to warp an image displayed on the display; and displayvia the display the virtual space and the virtual reality representationaccording to the spatial warping pattern.